Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add endpoints for backfilling history (MSC2716) #9247

Merged
merged 96 commits into from
Jun 22, 2021

Conversation

MadLittleMods
Copy link
Contributor

@MadLittleMods MadLittleMods commented Jan 28, 2021

Implement MSC2716 to add endpoints for backfilling history. This PR does not support federation use cases with the "marker" and "insertion" events.

For reviewers, it's probably best to see this in action with the associated Complement tests.


Complement MR: matrix-org/complement#68

TARDIS visualization MR: matrix-org/tardis#1

Getting started

The PR adds the POST /_matrix/client/unstable/org.matrix.msc2716/rooms/<roomID>/batch_send?prev_event=<eventID>&chunk_id=<chunkID> endpoint which can insert a chunk of events historically back in time next to the given prev_event. chunk_id comes from next_chunk_id in the response of the batch send endpoint and is derived from the "insertion" events added to each chunk. It's not required for the first batch send.

{
    "events": [ ... ],
    "state_events_at_start": [ ... ]
}

The /batchsend endpoint is behind a feature flag: experimental_features -> msc2716_enabled (defined in homseserver.yaml). And is only available to application services so you will need to add one to your homeserver.yml and use the as_token defined to interact with the API (other tokens will 403).

state_events_at_start is used to define the historical state events needed to auth the events like join events. These events will float outside of the normal DAG as outlier's and won't be visible in the chat history which also allows us to insert multiple chunks without having a bunch of @mxid joined the room noise between each chunk.

events is chronological(oldest to newest) chunk/list of events you want to insert. There is a reverse-chronological constraint on chunks so once you insert some messages, you can only insert older ones after that. tldr; Insert from your most recent history -> oldest history.

Why? depth is not re-calculated when historical messages are inserted into the DAG. This means we have to take care to insert in the right order. Events are sorted by (topological_ordering, stream_ordering) where topological_ordering is just depth. Normally, stream_ordering is an auto incrementing integer but for backfilled=true events, it decrements. Historical messages are inserted all at the same depth, and marked as backfilled so the stream_ordering decrements and each event is sorted behind the next. (from #9247 (comment))

If you're curious to look at a known working example, the Complement tests have barebones test cases interacting with this API, matrix-org/complement#68

Steps to reproduce:

  1. In your homeserver.yaml, add the feature flag to enable the /batchsend endpoint
    experimental_features:
      # Enable history backfilling support
      msc2716_enabled: true
  2. Define an application service in your homeserver.yaml. This could be one of your existing bridges. See the application service guide for an example of what the registration file would look like. We only care about the as_token in this case.
    app_service_config_files:
      - /data/my-as-registration.yaml
  3. POST /_matrix/client/unstable/org.matrix.msc2716/rooms/<roomID>/batch_send?prev_event=<eventID>&chunk_id=<chunkID> with the Authorization: Bearer <as_token> header and the following body:
    - prev_event is the event you want to insert next to. Your historical messages will appear after this event so pick one where the timestamp makes sense. To be a little more idiomatic for inserting historical events that happened before the Matrix room creation, prev_event could be some primordial creation event for the room.
    - chunk_id comes from next_chunk_id in the previous batch send response. It connects the last (most recent) message to the insertion event of the previous chunk. The parameter is not needed for your first chunk because there is nothing to connect to yet. Note: The messages will appear correctly on your local server without it but it's important to have this set for federated servers so messages backfill correctly.
    - You can change origin_server_ts in your events to whatever you want to display it as
    - Add a join event in state_events_at_start for any message author in the events
    - m.historical will automatically be added to each your events. This is important to mark them as backfilled, sort correctly, and skip the push notifications actions.
    {
        "events": [{
            "type": "m.room.message",
            "sender": "@maria:hs1",
            "origin_server_ts": 1620336731128,
            "content": {
                "msgtype": "m.text",
                "body": "Some historical message",
                "m.historical": true,
            },
        }],
        "state_events_at_start": [{
            "type": "m.room.member",
            "sender": "@maria:hs1",
            "origin_server_ts": 1620336731128,
            "content": {
                "membership": "join",
            },
            "state_key": "@maria:hs1",
        }]
    }

Dev notes

COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh

./scripts-dev/complement.sh

API:

Relevant code:

event_creation_handler.create_and_send_nonmember_event
EventCreationHandler.create_event
EventCreationHandler.create_new_client_event
EventBuilder.build
_generate_local_out_of_band_leave

persist_events or persist_event
_maybe_start_persisting
_persist_events
_persist_events_and_state_updates
_persist_events_txn
_update_metadata_tables_txn
_handle_mult_prev_events
-> simple_insert_many_txn `event_edges`

Call stacks for passing depth, prev_event_ids, auth_event_ids:

create_and_send_nonmember_event
create_event
create_new_client_event
builder.build
update_membership
update_membership_locked
_local_membership_update
create_event

COMPLEMENT_BASE_IMAGE=complement-synapse go test -tags msc2716 -v -count=1 ./tests/main_test.go ./tests/msc2716_test.go

Accessing the database

Access the sqlite database in the Docker container after Complement runs. Be sure to change the defer deployment.Destroy(t) call in the Complement tests to a defer time.Sleep(2 * time.Hour) so the Docker container stays alive after it's finished.

$ docker exec -it b91db4912057 /bin/bash
$ apt-get update
$ apt-get install sqlite3
$ sqlite3 /conf/homeserver.db

.tables
.schema events

select * from events where event_id='$LfK8zWi0g_snvqwi93vWGKj-iTf7gxWh_WRhTD5pALc';
select * from event_json where event_id='$LfK8zWi0g_snvqwi93vWGKj-iTf7gxWh_WRhTD5pALc';
select * from event_auth where event_id='$LfK8zWi0g_snvqwi93vWGKj-iTf7gxWh_WRhTD5pALc';
select * from event_auth_chains where event_id='$LfK8zWi0g_snvqwi93vWGKj-iTf7gxWh_WRhTD5pALc';
select * from event_auth_chain_to_calculate where event_id='$LfK8zWi0g_snvqwi93vWGKj-iTf7gxWh_WRhTD5pALc';

Event signing

test-event-signing.py (v1)

test-event-signing.py

from signedjson.key import (
    NACL_ED25519,
    decode_signing_key_base64,
    decode_verify_key_bytes,
    decode_verify_key_base64,
    generate_signing_key,
    get_verify_key,
    is_signing_algorithm_supported,
    read_signing_keys,
    write_signing_keys,
)
from signedjson.sign import (
    sign_json, verify_signed_json, SignatureVerifyException
)

SERVER_NAME = 'hs1'
KEY_ID = 'ed25519:a_imNW';
# This from the signature key_id `ed25519:a_FBvY`
VERSION = 'a_imNW'
SIGNING_KEY_BASE_64 = "vX2BK8l89Qk2EAVcaCiGiTUWG59dwleotTjzqu80C4w"
VERIFY_KEY_BASE_64 = "9d92WUgYwsKY0oWxOR1R61SJar9+D7uvz59IEDEjqyI"
JSON = {'my_key': 'my_data'}
# JSON = {
# 	'auth_events': ['$M7y6Yy1agkTx4eLXpZRndF1BdGk5pvCkjNCZkKlGIdY', '$ZtnAy9W445OtyeqCEk9Dcn6Nr14kYb4z-fJM8MwcVmI', '$gf4M_E3lR3QhumzTKMCncsDrTLZSVjuhcHZ4y6EtJzA', '$mSfCznaiy9xZvR7uQb7NdbNVS-1DNSe7lilB649H78g', '$8WTDdQw82NY8SUWUom53IjM1HzXiaB71oeU1mPjz7XY', '$ywbCJgiKeXtXuhN8OzxFD04-Esj89nxSt5miRxM5f78', '$ooT281Jg3nfzhjdE1rq0hUEOpu8sX1ublFUDYRPuyOk'], 'prev_events': ['$zWKxzzgcSVoLUDAeauzbOXkCTfrfNCcRQwEQToTZpof'],
# 	'type': 'm.room.member',
# 	'room_id': '!aqtKBNDKZSWCDMRpSH:hs1',
# 	'sender': '@maria:hs1',
# 	'content': {'membership': 'join'},
# 	'depth': 1,
# 	'prev_state': [],
# 	'state_key':
# 	'@maria:hs1',
# 	'origin': 'hs1', 
# 	'origin_server_ts': 1620167174120,
# 	'hashes': {'sha256': 'Cw+4O3jZVocrTLhdJ9vfoZq1SRYk4RB+/8X5EQ/xaXU'},
# 	'signatures': {'hs1': {'ed25519:a_imNW': '8/II10y7FO7nMBZLeF6aRMkDAkSjnRsJIu8oPGfG9AEvmvEvusNwe6klWfw7QPB0NfJpF5wyOd5rQrbX7ymcAg'}},
# 	'unsigned': {}
# }


#signing_key = generate_signing_key('zxcvb')
#verify_key = get_verify_key(signing_key)

signing_key = decode_signing_key_base64(NACL_ED25519, VERSION, SIGNING_KEY_BASE_64)
#verify_key = decode_verify_key_bytes(KEY_ID, VERIFY_KEY_BASE_64)
verify_key = decode_verify_key_base64(NACL_ED25519, VERSION, VERIFY_KEY_BASE_64)
verify_key_derived_from_signing_key = get_verify_key(signing_key)
if verify_key != verify_key_derived_from_signing_key:
	print("WARNING: verify_key and verify_key_derived_from_signing_key are different")

print(f"signing_key={signing_key}")
print(f"verify_key={verify_key} verify_key_derived_from_signing_key={verify_key_derived_from_signing_key}")

signed_json = sign_json(JSON, SERVER_NAME, signing_key)


try:
    verify_signed_json(signed_json, SERVER_NAME, verify_key)
    print('Signature is valid')
except SignatureVerifyException:
    print('Signature is invalid')
test-event-signing.py (v2)

test-event-signing.py

import copy
from signedjson.key import (
    NACL_ED25519,
    decode_signing_key_base64,
    decode_verify_key_bytes,
    decode_verify_key_base64,
    generate_signing_key,
    get_verify_key,
    is_signing_algorithm_supported,
    read_signing_keys,
    write_signing_keys,
)
from signedjson.sign import (
    sign_json, verify_signed_json, SignatureVerifyException
)

SERVER_NAME = 'hs1'
KEY_ID = 'ed25519:a_imNW';
# This from the signature key_id `ed25519:a_FBvY`
VERSION = 'a_imNW'
SIGNING_KEY_BASE_64 = "vX2BK8l89Qk2EAVcaCiGiTUWG59dwleotTjzqu80C4w"
VERIFY_KEY_BASE_64 = "9d92WUgYwsKY0oWxOR1R61SJar9+D7uvz59IEDEjqyI"


#signing_key = generate_signing_key('zxcvb')
#verify_key = get_verify_key(signing_key)

signing_key = decode_signing_key_base64(NACL_ED25519, VERSION, SIGNING_KEY_BASE_64)
#verify_key = decode_verify_key_bytes(KEY_ID, VERIFY_KEY_BASE_64)
verify_key = decode_verify_key_base64(NACL_ED25519, VERSION, VERIFY_KEY_BASE_64)
verify_key_derived_from_signing_key = get_verify_key(signing_key)
if verify_key != verify_key_derived_from_signing_key:
	print("WARNING: verify_key and verify_key_derived_from_signing_key are different")

print(f"signing_key={signing_key}")
print(f"verify_key={verify_key} verify_key_derived_from_signing_key={verify_key_derived_from_signing_key}")

#JSON = {'my_key': 'my_data'}
#signed_json = sign_json(JSON, SERVER_NAME, signing_key)
KNOWN_GOOD_JSON = {
	'auth_events': ['$M7y6Yy1agkTx4eLXpZRndF1BdGk5pvCkjNCZkKlGIdY'],
	'prev_events': ['$M7y6Yy1agkTx4eLXpZRndF1BdGk5pvCkjNCZkKlGIdY'],
	'type': 'm.room.member',
	'room_id': '!aqtKBNDKZSWCDMRpSH:hs1',
	'sender': '@the-bridge-user:hs1',
	'content': {'membership': 'join'},
	'depth': 2,
	'prev_state': [],
	'state_key': '@the-bridge-user:hs1',
	'origin': 'hs1',
	'origin_server_ts': 1620167173759,
	'hashes': {'sha256': 'FSyUP1X9VrlCCPztDLooSZSjcWqbUV2T7j7Z3o06zSw'},
	'signatures': {'hs1': {'ed25519:a_imNW': 'CO6WUNuoZb8bg0cH9zoywZWqzEc2YogZsp6jqhISjhIvv/HDMJYf0INlxYpo3m67xcXVYrh1LgeVw4qSRFWPDQ'}},
	'unsigned': {}
}
KNOWN_BAD_JSON = {
	'auth_events': ['$M7y6Yy1agkTx4eLXpZRndF1BdGk5pvCkjNCZkKlGIdY', '$ZtnAy9W445OtyeqCEk9Dcn6Nr14kYb4z-fJM8MwcVmI', '$gf4M_E3lR3QhumzTKMCncsDrTLZSVjuhcHZ4y6EtJzA', '$mSfCznaiy9xZvR7uQb7NdbNVS-1DNSe7lilB649H78g', '$8WTDdQw82NY8SUWUom53IjM1HzXiaB71oeU1mPjz7XY', '$ywbCJgiKeXtXuhN8OzxFD04-Esj89nxSt5miRxM5f78', '$ooT281Jg3nfzhjdE1rq0hUEOpu8sX1ublFUDYRPuyOk'],
	'prev_events': ['$zWKxzzgcSVoLUDAeauzbOXkCTfrfNCcRQwEQToTZpof'],
	'type': 'm.room.member',
	'room_id': '!aqtKBNDKZSWCDMRpSH:hs1',
	'sender': '@maria:hs1',
	'content': {'membership': 'join'},
	'depth': 1,
	'prev_state': [],
	'state_key':
	'@maria:hs1',
	'origin': 'hs1', 
	'origin_server_ts': 1620167174120,
	'hashes': {'sha256': 'Cw+4O3jZVocrTLhdJ9vfoZq1SRYk4RB+/8X5EQ/xaXU'},
	'signatures': {'hs1': {'ed25519:a_imNW': '8/II10y7FO7nMBZLeF6aRMkDAkSjnRsJIu8oPGfG9AEvmvEvusNwe6klWfw7QPB0NfJpF5wyOd5rQrbX7ymcAg'}},
	'unsigned': {}
}
# Switch around KNOWN_GOOD_JSON and KNOWN_BAD_JSON here
signed_json = KNOWN_BAD_JSON


copied_json = copy.deepcopy(signed_json)
double_checked_signed_json = sign_json(copied_json, SERVER_NAME, signing_key)
double_checked_signatures = double_checked_signed_json.get("signatures", {})
double_checked_server_signature = double_checked_signatures.get(SERVER_NAME, {}).get(KEY_ID)
print(f"double_check_signed_json={double_checked_signatures}")
server_signature = signed_json.get("signatures", {}).get(SERVER_NAME, {}).get(KEY_ID)
if double_checked_server_signature != server_signature:
	print(f"WANRING: When we re-signed the object and checked the signatures, they did NOT match!\ndouble_checked_server_signature={double_checked_server_signature}\nserver_signature={server_signature}")


try:
    verify_signed_json(signed_json, SERVER_NAME, verify_key)
    print('Signature is valid')
except SignatureVerifyException:
    print('Signature is invalid')

Todo

  • ts query param to override origin_server_ts
  • prev_event query param
  • Proper depth
  • m.historical event field
  • Add tests within Synapse

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
  • Pull request includes a sign off
  • Code style is correct (run the linters)

MadLittleMods added a commit to MadLittleMods/tardis that referenced this pull request Feb 2, 2021
Edits to make TARDIS work with Synapse while writing Complement tests for [MSC 2716](matrix-org/matrix-spec-proposals#2716).

 - matrix-org/synapse#9247
 - matrix-org/complement#68
synapse/events/utils.py Outdated Show resolved Hide resolved
TODO: Is the assumption of anytime we pass in prev_event_ids, we use same depth
good enough? What corner cases are there? I see that we also pass in prev_event_ids this in
synapse/handlers/room_member.py so need to make sure that still work as expected
@MadLittleMods MadLittleMods force-pushed the eric/msc2716-backfilling-history branch from 46625b7 to 9b5e057 Compare February 5, 2021 04:43
@MadLittleMods MadLittleMods requested a review from a team February 5, 2021 06:10
@clokep
Copy link
Member

clokep commented Feb 5, 2021

@MadLittleMods This seems to have some style / CI issues. Were you looking for general feedback or were you hoping to get this merged?

@MadLittleMods
Copy link
Contributor Author

MadLittleMods commented Feb 5, 2021

@clokep Some general comments and answer questions in the PR. What tests and where to add, etc?

What's more needed content wise to merge. Outside are a few lints to clean up.

Behind feature flag or unstable build type of thing?

@erikjohnston
Copy link
Member

On the complemenet tests: you may need to merge in latest develop into this branch and the same named branch on complement repo (or delete that branch)? The tests failures look to be mostly for things like spaces and knocking, which have recently had work done on them?

@MadLittleMods MadLittleMods requested review from erikjohnston and removed request for erikjohnston June 21, 2021 22:29
@MadLittleMods MadLittleMods changed the title Draft: Add endpoints for backfilling history (MSC2716) Add endpoints for backfilling history (MSC2716) Jun 21, 2021
Copy link
Member

@erikjohnston erikjohnston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@erikjohnston erikjohnston merged commit 96f6293 into develop Jun 22, 2021
@erikjohnston erikjohnston deleted the eric/msc2716-backfilling-history branch June 22, 2021 09:02
@MadLittleMods
Copy link
Contributor Author

Woot! Thank you @erikjohnston for all the help guiding this along and review to push this to the right places! 🐗

babolivier added a commit that referenced this pull request Jun 24, 2021
Synapse 1.37.0rc1 (2021-06-24)
==============================

This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface.

This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings.

Features
--------

- Implement "room knocking" as per [MSC2403](matrix-org/matrix-spec-proposals#2403). Contributed by @Sorunome and anoa. ([\#6739](#6739), [\#9359](#9359), [\#10167](#10167), [\#10212](#10212), [\#10227](#10227))
- Add experimental support for backfilling history into rooms ([MSC2716](matrix-org/matrix-spec-proposals#2716)). ([\#9247](#9247))
- Implement a generic interface for third-party plugin modules. ([\#10062](#10062), [\#10206](#10206))
- Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](#10108))
- Ensure that errors during startup are written to the logs and the console. ([\#10191](#10191))

Bugfixes
--------

- Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](#10115))
- Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](#10154))
- Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](#10157), [\#10158](#10158))
- Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](#10175))
- Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](#10184))
- Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](#10195))
- Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](#10208))
- Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](#10221))

Improved Documentation
----------------------

- Add a new guide to decoding request logs. ([\#8436](#8436))
- Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](#10122))
- Fix broken links in documentation. ([\#10180](#10180))
- Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](#10198))

Deprecations and Removals
-------------------------

- The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](#10062), [\#10210](#10210), [\#10238](#10238))
- Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](#10161))
- Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](#10194))

Internal Changes
----------------

- Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](#9933))
- Add type hints to the federation servlets. ([\#10080](#10080))
- Improve OpenTracing for event persistence. ([\#10134](#10134), [\#10193](#10193))
- Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](#10143))
- Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](#10144))
- Refactor EventPersistenceQueue. ([\#10145](#10145))
- Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](#10148))
- Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](#10155))
- Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](#10156))
- Fetch the corresponding complement branch when performing CI. ([\#10160](#10160))
- Add some developer documentation about boolean columns in database schemas. ([\#10164](#10164))
- Add extra logging fields to better debug where events are being soft failed. ([\#10168](#10168))
- Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](#10183))
- Improve comments in structured logging code. ([\#10188](#10188))
- Update [MSC3083](matrix-org/matrix-spec-proposals#3083) support with modifications from the MSC. ([\#10189](#10189))
- Remove redundant DNS lookup limiter. ([\#10190](#10190))
- Upgrade `black` linting tool to 21.6b0. ([\#10197](#10197))
- Expose OpenTracing trace id in response headers. ([\#10199](#10199))
Half-Shot added a commit that referenced this pull request Jun 28, 2021
Synapse 1.37.0rc1 (2021-06-24)
==============================

This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface.

This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings.

Features
--------

- Implement "room knocking" as per [MSC2403](matrix-org/matrix-spec-proposals#2403). Contributed by @Sorunome and anoa. ([\#6739](#6739), [\#9359](#9359), [\#10167](#10167), [\#10212](#10212), [\#10227](#10227))
- Add experimental support for backfilling history into rooms ([MSC2716](matrix-org/matrix-spec-proposals#2716)). ([\#9247](#9247))
- Implement a generic interface for third-party plugin modules. ([\#10062](#10062), [\#10206](#10206))
- Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](#10108))
- Ensure that errors during startup are written to the logs and the console. ([\#10191](#10191))

Bugfixes
--------

- Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](#10115))
- Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](#10154))
- Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](#10157), [\#10158](#10158))
- Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](#10175))
- Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](#10184))
- Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](#10195))
- Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](#10208))
- Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](#10221))

Improved Documentation
----------------------

- Add a new guide to decoding request logs. ([\#8436](#8436))
- Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](#10122))
- Fix broken links in documentation. ([\#10180](#10180))
- Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](#10198))

Deprecations and Removals
-------------------------

- The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](#10062), [\#10210](#10210), [\#10238](#10238))
- Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](#10161))
- Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](#10194))

Internal Changes
----------------

- Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](#9933))
- Add type hints to the federation servlets. ([\#10080](#10080))
- Improve OpenTracing for event persistence. ([\#10134](#10134), [\#10193](#10193))
- Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](#10143))
- Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](#10144))
- Refactor EventPersistenceQueue. ([\#10145](#10145))
- Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](#10148))
- Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](#10155))
- Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](#10156))
- Fetch the corresponding complement branch when performing CI. ([\#10160](#10160))
- Add some developer documentation about boolean columns in database schemas. ([\#10164](#10164))
- Add extra logging fields to better debug where events are being soft failed. ([\#10168](#10168))
- Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](#10183))
- Improve comments in structured logging code. ([\#10188](#10188))
- Update [MSC3083](matrix-org/matrix-spec-proposals#3083) support with modifications from the MSC. ([\#10189](#10189))
- Remove redundant DNS lookup limiter. ([\#10190](#10190))
- Upgrade `black` linting tool to 21.6b0. ([\#10197](#10197))
- Expose OpenTracing trace id in response headers. ([\#10199](#10199))
MadLittleMods added a commit that referenced this pull request Jun 29, 2021
Follow-up to #9247

Part of MSC2716: matrix-org/matrix-spec-proposals#2716

---

Previously, Synapse would throw a 403,
`Cannot force another user to join.`,
because we were trying to use `?user_id` from a single virtual user
which did not match with messages from other users in the chunk.
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Jul 1, 2021
Synapse 1.37.0 (2021-06-29)
===========================

This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface.

This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings.

Synapse 1.37.0rc1 (2021-06-24)
==============================

Features
--------

- Implement "room knocking" as per [MSC2403](matrix-org/matrix-spec-proposals#2403). Contributed by @Sorunome and anoa. ([\#6739](matrix-org/synapse#6739), [\#9359](matrix-org/synapse#9359), [\#10167](matrix-org/synapse#10167), [\#10212](matrix-org/synapse#10212), [\#10227](matrix-org/synapse#10227))
- Add experimental support for backfilling history into rooms ([MSC2716](matrix-org/matrix-spec-proposals#2716)). ([\#9247](matrix-org/synapse#9247))
- Implement a generic interface for third-party plugin modules. ([\#10062](matrix-org/synapse#10062), [\#10206](matrix-org/synapse#10206))
- Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](matrix-org/synapse#10108))
- Ensure that errors during startup are written to the logs and the console. ([\#10191](matrix-org/synapse#10191))


Bugfixes
--------

- Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](matrix-org/synapse#10115))
- Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](matrix-org/synapse#10154))
- Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](matrix-org/synapse#10157), [\#10158](matrix-org/synapse#10158))
- Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](matrix-org/synapse#10175))
- Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](matrix-org/synapse#10184))
- Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](matrix-org/synapse#10195))
- Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](matrix-org/synapse#10208))
- Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](matrix-org/synapse#10221))


Improved Documentation
----------------------

- Add a new guide to decoding request logs. ([\#8436](matrix-org/synapse#8436))
- Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](matrix-org/synapse#10122))
- Fix broken links in documentation. ([\#10180](matrix-org/synapse#10180))
- Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](matrix-org/synapse#10198))


Deprecations and Removals
-------------------------

- The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](matrix-org/synapse#10062), [\#10210](matrix-org/synapse#10210), [\#10238](matrix-org/synapse#10238))
- Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](matrix-org/synapse#10161))
- Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](matrix-org/synapse#10194))


Internal Changes
----------------

- Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](matrix-org/synapse#9933))
- Add type hints to the federation servlets. ([\#10080](matrix-org/synapse#10080))
- Improve OpenTracing for event persistence. ([\#10134](matrix-org/synapse#10134), [\#10193](matrix-org/synapse#10193))
- Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](matrix-org/synapse#10143))
- Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](matrix-org/synapse#10144))
- Refactor EventPersistenceQueue. ([\#10145](matrix-org/synapse#10145))
- Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](matrix-org/synapse#10148))
- Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](matrix-org/synapse#10155))
- Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](matrix-org/synapse#10156))
- Fetch the corresponding complement branch when performing CI. ([\#10160](matrix-org/synapse#10160))
- Add some developer documentation about boolean columns in database schemas. ([\#10164](matrix-org/synapse#10164))
- Add extra logging fields to better debug where events are being soft failed. ([\#10168](matrix-org/synapse#10168))
- Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](matrix-org/synapse#10183))
- Improve comments in structured logging code. ([\#10188](matrix-org/synapse#10188))
- Update [MSC3083](matrix-org/matrix-spec-proposals#3083) support with modifications from the MSC. ([\#10189](matrix-org/synapse#10189))
- Remove redundant DNS lookup limiter. ([\#10190](matrix-org/synapse#10190))
- Upgrade `black` linting tool to 21.6b0. ([\#10197](matrix-org/synapse#10197))
- Expose OpenTracing trace id in response headers. ([\#10199](matrix-org/synapse#10199))


Synapse 1.36.0 (2021-06-15)
===========================

No significant changes.


Synapse 1.36.0rc2 (2021-06-11)
==============================

Bugfixes
--------

- Fix a bug which caused  presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](matrix-org/synapse#10149))
- Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](matrix-org/synapse#10163))
- Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](matrix-org/synapse#10165))


Synapse 1.36.0rc1 (2021-06-08)
==============================

Features
--------

- Add new endpoint `/_matrix/client/r0/rooms/{roomId}/aliases` from Client-Server API r0.6.1 (previously [MSC2432](matrix-org/matrix-spec-proposals#2432)). ([\#9224](matrix-org/synapse#9224))
- Improve performance of incoming federation transactions in large rooms. ([\#9953](matrix-org/synapse#9953), [\#9973](matrix-org/synapse#9973))
- Rewrite logic around verifying JSON object and fetching server keys to be more performant and use less memory. ([\#10035](matrix-org/synapse#10035))
- Add new admin APIs for unprotecting local media from quarantine. Contributed by @dklimpel. ([\#10040](matrix-org/synapse#10040))
- Add new admin APIs to remove media by media ID from quarantine. Contributed by @dklimpel. ([\#10044](matrix-org/synapse#10044))
- Make reason and score parameters optional for reporting content. Implements [MSC2414](matrix-org/matrix-spec-proposals#2414). Contributed by Callum Brown. ([\#10077](matrix-org/synapse#10077))
- Add support for routing more requests to workers. ([\#10084](matrix-org/synapse#10084))
- Report OpenTracing spans for database activity. ([\#10113](matrix-org/synapse#10113), [\#10136](matrix-org/synapse#10136), [\#10141](matrix-org/synapse#10141))
- Significantly reduce memory usage of joining large remote rooms. ([\#10117](matrix-org/synapse#10117))


Bugfixes
--------

- Fixed a bug causing replication requests to fail when receiving a lot of events via federation. ([\#10082](matrix-org/synapse#10082))
- Fix a bug in the `force_tracing_for_users` option introduced in Synapse v1.35 which meant that the OpenTracing spans produced were missing most tags. ([\#10092](matrix-org/synapse#10092))
- Fixed a bug that could cause Synapse to stop notifying application services. Contributed by Willem Mulder. ([\#10107](matrix-org/synapse#10107))
- Fix bug where the server would attempt to fetch the same history in the room from a remote server multiple times in parallel. ([\#10116](matrix-org/synapse#10116))
- Fix a bug introduced in Synapse 1.33.0 which caused replication requests to fail when receiving a lot of very large events via federation. ([\#10118](matrix-org/synapse#10118))
- Fix bug when using workers where pagination requests failed if a remote server returned zero events from `/backfill`. Introduced in 1.35.0. ([\#10133](matrix-org/synapse#10133))


Improved Documentation
----------------------

- Clarify security note regarding hosting Synapse on the same domain as other web applications. ([\#9221](matrix-org/synapse#9221))
- Update CAPTCHA documentation to mention turning off the verify origin feature. Contributed by @aaronraimist. ([\#10046](matrix-org/synapse#10046))
- Tweak wording of database recommendation in `INSTALL.md`. Contributed by @aaronraimist. ([\#10057](matrix-org/synapse#10057))
- Add initial infrastructure for rendering Synapse documentation with mdbook. ([\#10086](matrix-org/synapse#10086))
- Convert the remaining Admin API documentation files to markdown. ([\#10089](matrix-org/synapse#10089))
- Make a link in docs use HTTPS. Contributed by @RhnSharma. ([\#10130](matrix-org/synapse#10130))
- Fix broken link in Docker docs. ([\#10132](matrix-org/synapse#10132))


Deprecations and Removals
-------------------------

- Remove the experimental `spaces_enabled` flag. The spaces features are always available now. ([\#10063](matrix-org/synapse#10063))


Internal Changes
----------------

- Tell CircleCI to build Docker images from `main` branch. ([\#9906](matrix-org/synapse#9906))
- Simplify naming convention for release branches to only include the major and minor version numbers. ([\#10013](matrix-org/synapse#10013))
- Add `parse_strings_from_args` for parsing an array from query parameters. ([\#10048](matrix-org/synapse#10048), [\#10137](matrix-org/synapse#10137))
- Remove some dead code regarding TLS certificate handling. ([\#10054](matrix-org/synapse#10054))
- Remove redundant, unmaintained `convert_server_keys` script. ([\#10055](matrix-org/synapse#10055))
- Improve the error message printed by synctl when synapse fails to start. ([\#10059](matrix-org/synapse#10059))
- Fix GitHub Actions lint for newsfragments. ([\#10069](matrix-org/synapse#10069))
- Update opentracing to inject the right context into the carrier. ([\#10074](matrix-org/synapse#10074))
- Fix up `BatchingQueue` implementation. ([\#10078](matrix-org/synapse#10078))
- Log method and path when dropping request due to size limit. ([\#10091](matrix-org/synapse#10091))
- In Github Actions workflows, summarize the Sytest results in an easy-to-read format. ([\#10094](matrix-org/synapse#10094))
- Make `/sync` do fewer state resolutions. ([\#10102](matrix-org/synapse#10102))
- Add missing type hints to the admin API servlets. ([\#10105](matrix-org/synapse#10105))
- Improve opentracing annotations for `Notifier`. ([\#10111](matrix-org/synapse#10111))
- Enable Prometheus metrics for the jaeger client library. ([\#10112](matrix-org/synapse#10112))
- Work to improve the responsiveness of `/sync` requests. ([\#10124](matrix-org/synapse#10124))
- OpenTracing: use a consistent name for background processes. ([\#10135](matrix-org/synapse#10135))
MadLittleMods added a commit that referenced this pull request Jul 7, 2021
Follow-up to #9247

Part of MSC2716: matrix-org/matrix-spec-proposals#2716

---

Previously, Synapse would throw a 403,
`Cannot force another user to join.`,
because we were trying to use `?user_id` from a single virtual user
which did not match with messages from other users in the chunk.
MadLittleMods added a commit that referenced this pull request Jul 7, 2021
Follow-up to #9247

Part of MSC2716: matrix-org/matrix-spec-proposals#2716

---

Previously, Synapse would throw a 403,
`Cannot force another user to join.`,
because we were trying to use `?user_id` from a single virtual user
which did not match with messages from other users in the chunk.
MadLittleMods added a commit that referenced this pull request Jul 13, 2021
)

Fix messages from multiple senders in historical chunk. This also means that an app service does not need to define `?user_id` when using this endpoint.

Follow-up to #9247

Part of MSC2716: matrix-org/matrix-spec-proposals#2716
MadLittleMods added a commit that referenced this pull request Jul 28, 2021
… any scrollback history (MSC2716) (#10245)

* Make historical messages available to federated servers

Part of MSC2716: matrix-org/matrix-spec-proposals#2716

Follow-up to #9247

* Debug message not available on federation

* Add base starting insertion point when no chunk ID is provided

* Fix messages from multiple senders in historical chunk

Follow-up to #9247

Part of MSC2716: matrix-org/matrix-spec-proposals#2716

---

Previously, Synapse would throw a 403,
`Cannot force another user to join.`,
because we were trying to use `?user_id` from a single virtual user
which did not match with messages from other users in the chunk.

* Remove debug lines

* Messing with selecting insertion event extremeties

* Move db schema change to new version

* Add more better comments

* Make a fake requester with just what we need

See #10276 (comment)

* Store insertion events in table

* Make base insertion event float off on its own

See #10250 (comment)

Conflicts:
	synapse/rest/client/v1/room.py

* Validate that the app service can actually control the given user

See #10276 (comment)

Conflicts:
	synapse/rest/client/v1/room.py

* Add some better comments on what we're trying to check for

* Continue debugging

* Share validation logic

* Add inserted historical messages to /backfill response

* Remove debug sql queries

* Some marker event implemntation trials

* Clean up PR

* Rename insertion_event_id to just event_id

* Add some better sql comments

* More accurate description

* Add changelog

* Make it clear what MSC the change is part of

* Add more detail on which insertion event came through

* Address review and improve sql queries

* Only use event_id as unique constraint

* Fix test case where insertion event is already in the normal DAG

* Remove debug changes

* Switch to chunk events so we can auth via power_levels

Previously, we were using `content.chunk_id` to connect one
chunk to another. But these events can be from any `sender`
and we can't tell who should be able to send historical events.
We know we only want the application service to do it but these
events have the sender of a real historical message, not the
application service user ID as the sender. Other federated homeservers
also have no indicator which senders are an application service on
the originating homeserver.

So we want to auth all of the MSC2716 events via power_levels
and have them be sent by the application service with proper
PL levels in the room.

* Switch to chunk events for federation

* Add unstable room version to support new historical PL

* Fix federated events being rejected for no state_groups

Add fix from #10439
until it merges.

* Only connect base insertion event to prev_event_ids

Per discussion with @erikjohnston,
https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$12bTUiObDFdHLAYtT7E-BvYRp3k_xv8w0dUQHibasJk?via=jki.re&via=matrix.org

* Make it possible to get the room_version with txn

* Allow but ignore historical events in unsupported room version

See #10245 (comment)

We can't reject historical events on unsupported room versions because homeservers without knowledge of MSC2716 or the new room version don't reject historical events either.

Since we can't rely on the auth check here to stop historical events on unsupported room versions, I've added some additional checks in the processing/persisting code (`synapse/storage/databases/main/events.py` ->  `_handle_insertion_event` and `_handle_chunk_event`). I've had to do some refactoring so there is method to fetch the room version by `txn`.

* Move to unique index syntax

See #10245 (comment)

* High-level document how the insertion->chunk lookup works

* Remove create_event fallback for room_versions

See https://github.com/matrix-org/synapse/pull/10245/files#r677641879

* Use updated method name
@MadLittleMods MadLittleMods added the T-Enhancement New features, changes in functionality, improvements in performance, or user-facing enhancements. label Jul 28, 2021
MadLittleMods added a commit that referenced this pull request Aug 4, 2021
* Make historical messages available to federated servers

Part of MSC2716: matrix-org/matrix-spec-proposals#2716

Follow-up to #9247

* Debug message not available on federation

* Add base starting insertion point when no chunk ID is provided

* Fix messages from multiple senders in historical chunk

Follow-up to #9247

Part of MSC2716: matrix-org/matrix-spec-proposals#2716

---

Previously, Synapse would throw a 403,
`Cannot force another user to join.`,
because we were trying to use `?user_id` from a single virtual user
which did not match with messages from other users in the chunk.

* Remove debug lines

* Messing with selecting insertion event extremeties

* Move db schema change to new version

* Add more better comments

* Make a fake requester with just what we need

See #10276 (comment)

* Store insertion events in table

* Make base insertion event float off on its own

See #10250 (comment)

Conflicts:
	synapse/rest/client/v1/room.py

* Validate that the app service can actually control the given user

See #10276 (comment)

Conflicts:
	synapse/rest/client/v1/room.py

* Add some better comments on what we're trying to check for

* Continue debugging

* Share validation logic

* Add inserted historical messages to /backfill response

* Remove debug sql queries

* Some marker event implemntation trials

* Clean up PR

* Rename insertion_event_id to just event_id

* Add some better sql comments

* More accurate description

* Add changelog

* Make it clear what MSC the change is part of

* Add more detail on which insertion event came through

* Address review and improve sql queries

* Only use event_id as unique constraint

* Fix test case where insertion event is already in the normal DAG

* Remove debug changes

* Add support for MSC2716 marker events

* Process markers when we receive it over federation

* WIP: make hs2 backfill historical messages after marker event

* hs2 to better ask for insertion event extremity

But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group`
error

* Add insertion_event_extremities table

* Switch to chunk events so we can auth via power_levels

Previously, we were using `content.chunk_id` to connect one
chunk to another. But these events can be from any `sender`
and we can't tell who should be able to send historical events.
We know we only want the application service to do it but these
events have the sender of a real historical message, not the
application service user ID as the sender. Other federated homeservers
also have no indicator which senders are an application service on
the originating homeserver.

So we want to auth all of the MSC2716 events via power_levels
and have them be sent by the application service with proper
PL levels in the room.

* Switch to chunk events for federation

* Add unstable room version to support new historical PL

* Messy: Fix undefined state_group for federated historical events

```
2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill
    await self.backfill(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill
    await self._auth_and_persist_event(dest, event, context, backfilled=True)
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event
    await self._run_push_actions_and_persist_event(event, context, backfilled)
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event
    await self.persist_events_and_notify(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify
    events, max_stream_token = await self.storage.persistence.persist_events(
  File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events
    ret_vals = await yieldable_gather_results(enqueue, partitioned.items())
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop
    ret = await self._per_item_callback(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch
    await self.persist_events_store._persist_events_and_state_updates(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise
    raise exception.with_traceback(traceback)
  File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn
    self._store_event_state_mappings_txn(txn, events_and_contexts)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn
    self.db_pool.simple_insert_many_txn(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn
    txn.execute_batch(sql, vals)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch
    self.executemany(sql, args)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany
    self._do_execute(self.txn.executemany, sql, *args)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute
    return func(sql, *args)
sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group
```

* Revert "Messy: Fix undefined state_group for federated historical events"

This reverts commit 187ab28.

* Fix federated events being rejected for no state_groups

Add fix from #10439
until it merges.

* Adapting to experimental room version

* Some log cleanup

* Add better comments around extremity fetching code and why

* Rename to be more accurate to what the function returns

* Add changelog

* Ignore rejected events

* Use simplified upsert

* Add Erik's explanation of extra event checks

See #10498 (comment)

* Clarify that the depth is not directly correlated to the backwards extremity that we return

See #10498 (comment)

* lock only matters for sqlite

See #10498 (comment)

* Move new SQL changes to its own delta file

* Clean up upsert docstring

* Bump database schema version (62)
babolivier added a commit to matrix-org/synapse-dinsic that referenced this pull request Sep 1, 2021
Synapse 1.37.0 (2021-06-29)
===========================

This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface.

This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings.

Synapse 1.37.0rc1 (2021-06-24)
==============================

Features
--------

- Implement "room knocking" as per [MSC2403](matrix-org/matrix-spec-proposals#2403). Contributed by @Sorunome and anoa. ([\#6739](matrix-org/synapse#6739), [\#9359](matrix-org/synapse#9359), [\#10167](matrix-org/synapse#10167), [\#10212](matrix-org/synapse#10212), [\#10227](matrix-org/synapse#10227))
- Add experimental support for backfilling history into rooms ([MSC2716](matrix-org/matrix-spec-proposals#2716)). ([\#9247](matrix-org/synapse#9247))
- Implement a generic interface for third-party plugin modules. ([\#10062](matrix-org/synapse#10062), [\#10206](matrix-org/synapse#10206))
- Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](matrix-org/synapse#10108))
- Ensure that errors during startup are written to the logs and the console. ([\#10191](matrix-org/synapse#10191))

Bugfixes
--------

- Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](matrix-org/synapse#10115))
- Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](matrix-org/synapse#10154))
- Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](matrix-org/synapse#10157), [\#10158](matrix-org/synapse#10158))
- Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](matrix-org/synapse#10175))
- Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](matrix-org/synapse#10184))
- Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](matrix-org/synapse#10195))
- Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](matrix-org/synapse#10208))
- Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](matrix-org/synapse#10221))

Improved Documentation
----------------------

- Add a new guide to decoding request logs. ([\#8436](matrix-org/synapse#8436))
- Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](matrix-org/synapse#10122))
- Fix broken links in documentation. ([\#10180](matrix-org/synapse#10180))
- Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](matrix-org/synapse#10198))

Deprecations and Removals
-------------------------

- The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](matrix-org/synapse#10062), [\#10210](matrix-org/synapse#10210), [\#10238](matrix-org/synapse#10238))
- Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](matrix-org/synapse#10161))
- Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](matrix-org/synapse#10194))

Internal Changes
----------------

- Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](matrix-org/synapse#9933))
- Add type hints to the federation servlets. ([\#10080](matrix-org/synapse#10080))
- Improve OpenTracing for event persistence. ([\#10134](matrix-org/synapse#10134), [\#10193](matrix-org/synapse#10193))
- Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](matrix-org/synapse#10143))
- Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](matrix-org/synapse#10144))
- Refactor EventPersistenceQueue. ([\#10145](matrix-org/synapse#10145))
- Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](matrix-org/synapse#10148))
- Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](matrix-org/synapse#10155))
- Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](matrix-org/synapse#10156))
- Fetch the corresponding complement branch when performing CI. ([\#10160](matrix-org/synapse#10160))
- Add some developer documentation about boolean columns in database schemas. ([\#10164](matrix-org/synapse#10164))
- Add extra logging fields to better debug where events are being soft failed. ([\#10168](matrix-org/synapse#10168))
- Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](matrix-org/synapse#10183))
- Improve comments in structured logging code. ([\#10188](matrix-org/synapse#10188))
- Update [MSC3083](matrix-org/matrix-spec-proposals#3083) support with modifications from the MSC. ([\#10189](matrix-org/synapse#10189))
- Remove redundant DNS lookup limiter. ([\#10190](matrix-org/synapse#10190))
- Upgrade `black` linting tool to 21.6b0. ([\#10197](matrix-org/synapse#10197))
- Expose OpenTracing trace id in response headers. ([\#10199](matrix-org/synapse#10199))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
T-Enhancement New features, changes in functionality, improvements in performance, or user-facing enhancements.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants